Reusable Software Infrastructure for Stream Processing

Developers increasingly use streaming languages to write their data processing applications. Whilea variety of streaming languages exist, each targeting a particular application domain, they are allsimilar in that they represent a program as a graph of streams (i.e. sequences of data items) andoperators (i.e. data transformers). They are also similar in that they must process large volumesof data with high throughput. To meet this requirement, compilers of streaming languages mustprovide a variety of streaming-specific optimizations, including automatic parallelization. Tradi-tionally, when many languages share a set of optimizations, language implementors translate thesource languages into a common representation called an intermediate language (IL). Becauseoptimizations can modify the IL directly, they can be re-used by all of the source languages,reducing the overall engineering effort. However, traditional ILs and their associated optimiza-tions target single-machine, single-process programs. In contrast, the kinds of optimizations thatcompilers must perform in the streaming domain are quite different, and often involve reasoningacross multiple machines. Consequently, existing ILs are not suited to streaming languages.This thesis addresses the problem of how to provide a reusable infrastructure for stream pro-cessing languages. Central to the approach is the design of an intermediate language specificallyfor streaming languages and optimizations. The hypothesis is that an intermediate languagedesigned to meet the requirements of stream processing can assure implementation correctness;reduce overall implementation effort; and serve as a common substrate for critical optimizations.In evidence, this thesis provides the following contributions: (1) a catalog of common streamingoptimizations that helps define the requirements of a streaming IL; (2) a calculus that enablesreasoning about the correctness of source language translation and streaming optimizations; and(3) an intermediate language that preserves the semantics of the calculus, while addressing theimplementation issues omitted from the calculus. This work significantly reduces the effort ittakes to develop stream processing languages by making optimizations reusable across languages,and jump-starts innovation in language and optimization design.